Hash table entries insertion method and apparatus using virtual buckets

ABSTRACT

The present disclosure describes a process and apparatus for improving insertions of entries into a hash table. A large number of smaller virtual buckets may be combined together and associated with buckets used for hash table entry lookups and/or entry insertion. On insertion of an entry, hash table entries associated with a hashed-to virtual bucket may be moved between groups of buckets associated with the virtual bucket, to better distribute entries across the available buckets to reduce the number of entries in the largest buckets and the standard deviation of the bucket sizes across the entire hash table.

INCORPORATION BY REFERENCE

This application is a non-provisional application of U.S. provisionalapplication 62/410,284 entitled “Hash Table Entries Rebalancing Methodand Apparatus Using Virtual Buckets” filed Oct. 19, 2016. TheSpecification of U.S. provisional application 62/410,284 is hereby fullyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates to the fields of computing andnetworking. In particular, the present disclosure is related toprocesses and apparatuses for rebalancing hash table entries usingvirtual buckets.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart by inclusion in this section.

Various computing/networking applications involve flow classificationsand/or hash table lookups. For example, as the telecommunicationindustry transitions to software defined network (SDN) or networkfunction virtualization (NFV) to support legacy and upcoming usagemodels such as 5G on standard high volume servers, or the cloudcomputing industry, scalable and distributed software routing/switchinghas become one of the key requirements of the packet processingarchitecture. Two level distributed hashing architecture has beendeveloped to achieve high performance packet switching/routing usingeither a cluster of standard high-volume servers or a single systemcomposed of multiple CPU cores. In such architecture, network packetsare identified in part by lookup in various hash tables. Whether it istwo level or one level hashing, the performance of hash table degradesas the number of elements in each hash “bucket” increases.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be readily understood by the following detaileddescription in conjunction with the accompanying drawings. To facilitatethis description, like reference numerals designate like structuralelements. Embodiments are illustrated by way of example and not by wayof limitation in the figures of the accompanying drawings.

FIG. 1 illustrates an example computing/networking device having thehash table entries rebalancing technology of the present disclosure, inaccordance with various embodiments.

FIGS. 2-4 illustrate an example implementation of the hash table entriesrebalancing mechanism of the present disclosure, according to someembodiments.

FIG. 5 illustrates an example process for rebalancing entries in a hashtable using virtual buckets, in accordance with various embodiments.

FIG. 6 illustrates a block diagram of an example architecture of acomputing/networking device suitable for use to practice the presentdisclosure, in accordance with various embodiments.

FIG. 7 illustrates an example computer-readable storage medium withinstructions configured to enable a computing/networking device topractice aspects of the present disclosure, in accordance with variousembodiments.

DETAILED DESCRIPTION

In legacy hash table implementations, as more entries fill up eachbucket of the hash table it takes longer to find an available bucket toinsert a new entry into the table. This results in decreased insertionperformance. In legacy implementations, when entries are insertedrandomly into a fixed number of buckets (as is often the case with realworkloads), the number of entries into those buckets tend to form anormal distribution with some buckets containing far more entries thanthe average, some almost empty, and the rest clustered around theaverage.

This disclosure addresses these problems by providing a mechanism bywhich the entries in the buckets can be rebalanced at insertion time byadding only O(1) operations. This may have the effect of transferringentries from “more full” to “less full” buckets, reducing the maximumnumber of entries in any one bucket. This may significantly improveperformance, particularly when the hash table is highly sensitive tobucket size.

In the following description, various aspects of the illustrativeimplementations are described using terms commonly employed by thoseskilled in the art to convey the substance of their work to othersskilled in the art. However, it will be apparent to those skilled in theart that embodiments of the present disclosure may be practiced withonly some of the described aspects. For purposes of explanation,specific numbers, materials, and configurations are set forth in orderto provide a thorough understanding of the illustrative implementations.However, it will be apparent to one skilled in the art that embodimentsof the present disclosure may be practiced without the specific details.In other instances, well-known features are omitted or simplified inorder not to obscure the illustrative implementations.

In the following description, reference is made to the accompanyingdrawings that form a part hereof, wherein like numerals designate likeparts throughout, and in which is shown by way of illustrationembodiments in which the subject matter of the present disclosure may bepracticed. It is to be understood that other embodiments may be utilizedand structural or logical changes may be made without departing from thescope of the present disclosure. Therefore, the following detaileddescription is not to be taken in a limiting sense, and the scope ofembodiments is defined by the appended claims and their equivalents.

For the purposes of the present disclosure, the phrase “A and/or B”means (A), (B), or (A and B). For the purposes of the presentdisclosure, the phrase “A, B, and/or C” means (A), (B), (C), (A and B),(A and C), (B and C), or (A, B, and C).

The description may use perspective-based descriptions such astop/bottom, in/out, over/under, and the like. Such descriptions aremerely used to facilitate the discussion and are not intended torestrict the application of embodiments described herein to anyparticular orientation.

The description may use the phrases “in an embodiment,” or “inembodiments,” which may each refer to one or more of the same ordifferent embodiments. Furthermore, the terms “including,” “having,” andthe like, as used with respect to embodiments of the present disclosure,are synonymous.

The terms “coupled with” and “coupled to” and the like may be usedherein. “Coupled” may mean one or more of the following. “Coupled” maymean that two or more elements are in direct physical or electricalcontact. However, “coupled” may also mean that two or more elementsindirectly contact each other, but yet still cooperate or interact witheach other, and may mean that one or more other elements are coupled orconnected between the elements that are said to be coupled with eachother. By way of example and not limitation, “coupled” may mean two ormore elements or devices are coupled by electrical connections on aprinted circuit board such as a motherboard, for example. By way ofexample and not limitation, “coupled” may mean two or moreelements/devices cooperate and/or interact through one or more networklinkages such as wired and/or wireless networks. By way of example andnot limitation, a computing apparatus may include two or more computingdevices “coupled” on a motherboard or by one or more network linkages.

The term “module” in general, “insertion module” in particular, mayrefer to, be part of, or include an Application Specific IntegratedCircuit (ASIC), an electronic circuit, a combinatorial circuit, such asfield programmable gate array (FPGA) programmed with the implementationlogic, a processor (shared, dedicated, or group) and/or memory (shared,dedicated, or group) that execute one or more software or firmwareprograms with implementation logic, and/or other suitable componentsthat provide the described functionality with references to FIGS. 1-5below.

The term “computer-readable storage media” may refer to, be a part of,or otherwise include media on which data, including instructions of amodule (e.g., the insertion module described below) that may beexecuted, may reside. Computer-readable storage media may be eithertransitory, or non-transitory.

Various operations are described as multiple discrete operations inturn, in a manner that is most helpful in understanding the claimedsubject matter. However, the order of description should not beconstrued as to imply that these operations are necessarily orderdependent.

To improve hash table performance, a large number of smaller virtualbuckets may be combined together and associated with buckets used forhash table entry lookups and/or entry insertion. In this way, entriesmay be moved between groups of buckets at entry insertion time whilesubstantially maintaining the performance of the underlying hash table.

For example, a hash table may be sized to hold 2²¹ entries (a littleover 2 million). The table may be made up of 2¹⁶ (65,536) buckets thatmay each store 2⁵ (32) items. In addition, 2²⁰ virtual buckets may beused, with 2⁴ (16) virtual buckets associated for each real bucket.

Thus, when randomly inserting entries each virtual bucket may have onaverage 2 entries, with some being larger and smaller again following astatistical normal distribution. As (non-virtual) buckets start to gettoo full, some virtual buckets may be subtracted from a too-full bucketand added to another bucket that is less full to balance the number ofentries per bucket. This is in contrast to legacy hash tableimplementations that do not use virtual buckets, for example where anentry may be hashed and then 16 bits of the hash may be used to addressone of 2¹⁶ buckets. Statistically, if random keys are inserted in thisfashion, it is statistically likely that 30% of 2 million may beinserted before one of those buckets exceeds 32 items and the insertfails.

In embodiments, implementation of the systems and processes herein willbetter distribute entries across the available buckets and reduce thenumber of entries in the largest buckets and the standard deviation ofthe bucket sizes across the entire hash table.

FIG. 1 illustrates an example computing/networking device having thehash table entries rebalancing technology of the present disclosure, inaccordance with various embodiments. A computing/networking device 100may include an application or networking function 102. In embodiments,this may include an elastic flow distributor (EFD) of an SDN/NFD. Inembodiments, the application or networking function 102 may include anumber of hash tables 104. In embodiments, the hash tables 104 mayinclude an array of M buckets 106, and virtual buckets 108.Additionally, the application or networking function 102 may furtherinclude a mapping array 109, and an insertion module 110 associated withthe hash table 104 for inserting and rebalancing the hash table 104. Aninsertion module 110 may also include a hash function. The insertionmodule 110 may take an entry as input, compute deterministically thevirtual bucket index where that entry may be stored, and adjust thedistribution of virtual buckets among buckets to balance the number ofentries stored in the buckets.

In embodiments, buckets 106 may contain an array of entries, which mayalso be referred to as an array of keys. In embodiments, the arrayassociated with the buckets 106 may contain a fixed number of entries(N). In embodiments, an entry may include a network packet, anidentification for a network packet, or data associated with the networkpacket. In embodiments, an entry may be any other item information thatmay wish to be stored and/or retrieved from a hash table. Inembodiments, each of the buckets 106 may include a counter indicatingthe number of keys, or entries, that are stored in that bucket.

In embodiments, one or more virtual buckets 108 may be mapped to one ormore of the buckets 106, which may also be referred to as a group ofbuckets. Virtual buckets 108 may contain a choice that may indicate towhich group of buckets 106 a particular virtual bucket may beassociated. In embodiments, the mapping array 109 may be an array, orother data structure, that identifies the choice of each virtual bucketand maps the relationship between virtual buckets 108 and groups ofbuckets 106.

In embodiments, computing or networking device 100 may be any one ofsuch devices known in the art, e.g., servers, routers, switches,gateways, and so forth. In particular, in embodiments, computing ornetworking device 100 may be one of a plurality of networking devicesimplementing the modular forwarding table scalability technology of U.S.patent application Ser. No. 14/750,918, filed on Jun. 25, 2015, entitled“TECHNOLOGIES FOR MODULAR FORWARDING TABLE SCALABILITY,” whichspecification is hereby fully incorporated by reference.

FIGS. 2-4 illustrate an example implementation of the hash table entriesrebalancing mechanism of the present disclosure, according to someembodiments.

FIG. 2 illustrates an example implementation of a hash table that uses256 (0-255) virtual buckets 215 each having a virtual bucket identifier216 and a number of entries (keys) 217 associated with the virtualbucket. In embodiments, a hash function applied to an entry may producean identifier 216 for a virtual bucket (e.g., 215 a) to which that entrymay be assigned. In embodiments, a virtual bucket may have no entries(e.g., 215 d) or multiple entries (e.g., 215 a or 215 b), as reflectedby the number 217 for the virtual bucket.

In the example, there are 64 (0-63) buckets 218. Each respective virtualbucket 215 a, 215 b and so forth may have four choices for each of the64 buckets with which it may be associated, as may be indicated by themultiple arrows shown. In embodiments, there may be a different numberof choices for each of the respective virtual buckets 215. A mappingarray, such as mapping array 109 of FIG. 1, may be used to identify thebuckets associated with each virtual bucket. In embodiments, a function(interpolation) may be used to identify the buckets associated with eachvirtual bucket. This may include, for example, using an offset andmultiplication based on an initial number, using a fixed pseudorandomshuffle array, or some other suitable algorithm.

In the example implementation, a bucket identifier may be indicated inthe upper left corner, for example, bucket “0” 222 a and bucket “1” 222b. For each virtual bucket 215 there may be a group of buckets 218 towhich a virtual bucket may be associated. This information may berecorded in a mapping array as discussed above. This exampleimplementation includes, for virtual bucket “0” 215 a a mapping tobucket “0” 218 a. For virtual bucket “2” 215 b a mapping to buckets “1”218 b, “2” 218 c, “4” 218 d, and “6” 218 e. Consequently, bucket “1” 218b is associated with virtual buckets “2” 215 b and “7” 215 c. The totalnumber of entries associated with a bucket 218 a may be indicated in thelower left corner of the bucket, for example the number “4” 224 a. Thisnumber for a particular bucket may be the sum of each of the entriesassociated with each of the virtual buckets 215 associated with thebucket. For example, “5” 224 b is the sum of the number of entries ofvirtual bucket 2 (3) and virtual bucket 7 (2).

FIG. 3 illustrates an example implementation of an insertion of an entryinto the hash table. For example, an entry to insert 326 may have partof its data used as input to a hash function 328 that may result in ahash value 330. In this example, the hash value 330 corresponds tovirtual bucket “2” 215 b that already has 3 entries associated with it217 b. Virtual bucket “2” 215 b is associated with four buckets 218 b,218 c, 218 d, 218 e.

The four buckets 218 b, 218 c, 218 d, 218 e associated with virtualbucket “2” 215 b are examined to determine whether rebalancing may beneeded. The current entry counts 224 b, 224 c, 224 d, 224 e for each ofthe four buckets 218 b, 218 c, 218 d, 218 e are compared to determinewhich bucket has the lowest number of entries. In this example, bucket“4” 218 d has an entry count of “1” 224 d, which is lower than “5” 224b, “9” 224 c or “5” 224 e. Therefore, rebalancing of the hash table mayproceed by moving the entries associated with virtual bucket “2” 215 bfrom bucket “1” 218 b to bucket “4” 218 d. After rebalancing, the entry326 may be inserted into bucket “4” 218 d. In embodiments, the insertionmay happen first, for example into bucket “1” 218 b, and thenrebalancing performed after insertion.

FIG. 4 illustrates an example implementation of rebalancing based on aninsertion of an entry into the hash table. The four entries of virtualbucket “2” 415 b, which may be similar to 215 b of FIG. 3, but nowincremented by one to account for the inserted entry, is associated withbucket “1” 418 b, which may be similar to bucket “1” 218 b of FIG. 3,have been moved to bucket “4” 418 d, which may be similar to bucket “4”218 d of FIG. 3. As a result, the bucket “1” count 424 b, which may besimilar to 224 b of FIG. 3, is now 2 (5−3), and the bucket “4” count 424d, which may be similar to 224 d of FIG. 3, is now 5 (the original 1entry plus the 3 entries moved plus the 1 new entry inserted).

FIG. 5 illustrates an example process for rebalancing entries in a hashtable using virtual buckets, in accordance with various embodiments. Theprocess 500 may be performed, for example, by the device 100 of FIG. 1or the system 600 (e.g., computing device) configured to implement theinsertion module 650, similar to insertion module 110 and/or the hashingfunction module 852, similar to hashing function 328, described inreference to FIGS. 1-5.

The process 500 may begin at block 502, and may include receiving a datapacket for insertion as an entry into a hash table having a plurality ofbuckets divided into groups and the groups of buckets correspondinglyassociated with the plurality of virtual buckets. In embodiments, thismay include a plurality of buckets such as buckets 218 of FIG. 2. Inembodiments, the buckets may include a number of entries, for exampleentries of data packets. There may be a fixed number of buckets (M). Thebuckets may include a fixed number of slots for entries (N). Theplurality of virtual buckets, such as virtual buckets 215 of FIG. 2, mayeach mapped to a group of one or more buckets. In embodiments, becausethe number of virtual buckets 215 outnumbers the number M of buckets218, a hashing function, for example as implemented by the hashingfunction module 652 of FIG. 6 or 328 of FIG. 3, may distribute hashedentries over the virtual buckets with smaller maximum bucket size thanmay be achieved if the hashing function mapped entries directly into thelimited number of buckets. In embodiments, an entry may include all or aportion of the data packet, or may be some other data item that is to bestored in a hash table and later retrieved.

At block 504, the process 500 may include applying a hash function tothe data packet to determine a virtual bucket associated with the entry.In embodiments, the hash function may be implemented by the hashfunction module 652 of FIG. 6 or 328 of FIG. 3. In embodiments, the hashfunction may be one of any suitable hash functions, where the results ofthe hash function are to identify a virtual bucket to which the entrymay be associated.

At block 506, the process 500 may include selecting a first bucket of agroup of buckets associated with the virtual bucket. In embodiments,this may include the functionality described with respect to FIG. 3 orthe insertion module 650 of FIG. 6. In embodiments, there may be adefault bucket within the group of buckets for a virtual bucketidentified as the first bucket. For example, this may be the buckethaving the lowest identification number, such as bucket 218 b of FIG. 3.In embodiments, some other suitable algorithm may be used to select thefirst bucket. In embodiments, a mapping array, such as mapping array 109of FIG. 1, may be used to identify the virtual bucket and to identifyone or more buckets, or a group of buckets, that are associated with thevirtual bucket. In addition, the mapping array 109 may be used todetermine the default bucket or first bucket associated with the virtualbucket. In embodiments, the computational complexity of the actions ofthis block may involve O(1) additional time for the indirect action ofhashing to a virtual bucket in order to determine a bucket.

At block 508, the process 500 may include comparing respective countersof the group of buckets, wherein the respective counters indicate anumber of entries associated with each bucket of the group of buckets.In embodiments, the counters may be similar to a counter 224 aassociated with bucket 218 a of FIG. 2. In embodiments, the counter fora bucket may count entries associated with multiple virtual buckets thatare associated with the bucket.

At block 510, the process 500 may include determining, based on thecomparison, a second bucket of the group of buckets having a lowercounter value. In embodiments, the second bucket of the group of bucketsmay correspond to bucket “4” 218 d of FIG. 2, having a counter value of1 224 d. In embodiments, the computational complexity of the actions ofblocks 508 and/or 510 may involve O(1) additional time to scan throughgroups of buckets to select a second bucket.

At block 512, the process 500 may include moving one or more entriesassociated with the virtual bucket from the first bucket to the secondbucket. In embodiments, the result of this move may be seen by the twoentries associated with virtual bucket “2” 415 a moved from bucket “1”418 b to bucket “4” 418 d. In embodiments, the entries associated withother virtual buckets that are associated with bucket “1” 418 b are notmoved. In embodiments, all of the entries associated with the virtualbucket “2” 415 a may be moved to the second bucket “4” 418 d. Inembodiments, the counts associated with each bucket may be updatedaccordingly. In embodiments, the computational complexity of the actionsof this block may involve O(1) additional time for moving entries fromthe first to the second bucket.

At block 514, the process 500 may include inserting the entry into thesecond bucket. In embodiments, the result of this insertion may be seenby incrementing the entry count of the counter 424 d associated withbucket “4” 418 d, as well as incrementing the element count of virtualbucket “2” 417 a from 3 to 4 as shown in FIG. 4. In embodiments, thecomputational complexity of the actions of this block may involve O(1)additional time for updating counters.

It should be understood that the actions described in reference toprocess 500 may not necessarily occur in the described sequence. Inaddition, some actions may be added or omitted.

FIG. 6 illustrates a block diagram of an example architecture of acomputing/networking device suitable for use to practice the presentdisclosure, in accordance with various embodiments. As shown, computingdevice 600 may include one or more processors 602, each having one ormore processor cores, and system memory 604. The processor 602 mayinclude any type of unicore or multi-core processors. Each processorcore may include a central processing unit (CPU), and one or more levelof caches. The processor 602 may be implemented as an integratedcircuit. The computing device 600 may include mass storage devices 606(such as diskette, hard drive, volatile memory (e.g., dynamic randomaccess memory (DRAM)), compact disc read only memory (CD-ROM), digitalversatile disk (DVD) and so forth). In general, system memory 604 and/ormass storage devices 606 may be temporal and/or persistent storage ofany type, including, but not limited to, volatile and non-volatilememory, optical, magnetic, and/or solid state mass storage, and soforth. Volatile memory may include, but not be limited to, static and/ordynamic random access memory. Non-volatile memory may include, but notbe limited to, electrically erasable programmable read only memory,phase change memory, resistive memory, and so forth.

The computing device 600 may further include input/output (I/O) devices608 such as a display, keyboard, cursor control, remote control, gamingcontroller, image capture device, one or more three-dimensional camerasused to capture images, and so forth, and communication interfaces 610(such as network interface cards, modems, infrared receivers, radioreceivers (e.g., Bluetooth), and so forth). I/O devices 608 may besuitable for communicative connections with user devices or other systemdevices. In some embodiments, I/O devices 608 when used as user orsystem devices may include a device necessary for implementing thefunctionalities of receiving a data packet for insertion as an entry asdescribed in reference to FIG. 5.

The communication interfaces 610 may include communication chips (notshown) that may be configured to operate the device 600 in accordancewith a Global System for Mobile Communication (GSM), General PacketRadio Service (GPRS), Universal Mobile Telecommunications System (UMTS),High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or Long TermEvolution (LTE) network. The communication chips may also be configuredto operate in accordance with Enhanced Data for GSM Evolution (EDGE),GSM EDGE Radio Access Network (GERAN), Universal Terrestrial RadioAccess Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communicationchips may be configured to operate in accordance with Code DivisionMultiple Access (CDMA), Time Division Multiple Access (TDMA), DigitalEnhanced Cordless Telecommunications (DECT), Evolution-Data Optimized(EV-DO), derivatives thereof, as well as any other wireless protocolsthat are designated as 3G, 4G, 5G, and beyond. The communicationinterfaces 610 may operate in accordance with other wireless protocolsin other embodiments.

The above-described computing device 600 elements may be coupled to eachother via system bus 612, which may represent one or more buses. In thecase of multiple buses, they may be bridged by one or more bus bridges(not shown). Each of these elements may perform its conventionalfunctions known in the art. In particular, system memory 604 and massstorage devices 606 may be employed to store a working copy and apermanent copy of the programming instructions implementing theoperations and functionalities associated with FIG. 1 and/or FIG. 5,generally shown as computational logic 622. Computational logic 622 maybe implemented by assembler instructions supported by processor(s) 602or high-level languages that may be compiled into such instructions.

In embodiments, the Computational Logic 622 may contain an insertionmodule 650, which may perform one or more of the functions associatedwith FIGS. 1-5. Computational Logic 822 may contain a hash functionmodule 652, which may perform one or more of the hash functionsassociated with FIG. 1-5.

The permanent copy of the programming instructions may be placed intomass storage devices 606 in the factory, or in the field, though, forexample, a distribution medium (not shown), such as a compact disc (CD),or through communication interfaces 610 (from a distribution server (notshown)).

In embodiment, computing device 600 may be a wearable device, asmartphone, a computing tablet, a laptop computer, a desktop computer, aserver, a set-top box, a game console, a router, a switch, a gateway, orother networking equipment.

FIG. 7 illustrates an example computer-readable storage medium withinstructions configured to enable a computing/networking device topractice aspects of the present disclosure, in accordance with variousembodiments. As illustrated, non-transitory computer-readable storagemedium 702 may include a number of programming instructions 704 (e.g.,including insertion module 650 and hashing function module 652).Programming instructions 704 may be configured to enable a device, e.g.,computing device 600, in response to execution of the programminginstructions, to perform one or more operations of the processesdescribed in reference to FIGS. 1-5. In alternate embodiments,programming instructions 704 may be disposed on multiple non-transitorycomputer-readable storage media 702 instead. In still other embodiments,programming instructions 704 may be encoded in transitorycomputer-readable signals.

Referring also to FIG. 6, for some embodiments, processor 602 may bepackaged together with a computer-readable storage medium havingprogramming instructions 704 configured to practice all or selectedaspects of the hash table insertion and rebalancing related operationsearlier described. For one embodiment, processor 602 may be packagedtogether with a computer-readable storage medium having programminginstructions 704 to form a System in Package (SiP). For one embodiment,processor 602 may be integrated on the same die with a computer-readablestorage medium having programming instructions 704. For one embodiment,processor 602 may be packaged together with a computer-readable storagemedium having programming instructions 704 to form a System on Chip(SoC).

The corresponding structures, material, acts, and equivalents of allmeans or steps plus function elements in the claims below are intendedto include any structure, material or act for performing the function incombination with other claimed elements are specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill without departingfrom the scope and spirit of the disclosure. The embodiment was chosenand described in order to best explain the principles of the disclosureand the practical application, and to enable others of ordinary skill inthe art to understand the disclosure for embodiments with variousmodifications as are suited to the particular use contemplated.

EXAMPLES

Examples, according to various embodiments, may include the following.

Example 1 may be an apparatus for computing, comprising: one or morecomputer processors; a storage device coupled to the one or morecomputer processors; an insertion module communicatively coupled to theone or more processors, to manage insertion of entries into a hash tableresiding on the storage device, wherein the hash table has a pluralityof buckets divided into groups and the groups of buckets arecorrespondingly associated with a plurality of virtual buckets; andwherein the insertion module is to: receive a data packet for insertionas an entry into the hash table; apply a hash function to the datapacket to determine a virtual bucket associated with the entry; select afirst bucket of the group of buckets associated with the virtual bucket;compare respective counters of the group of buckets, wherein therespective counters indicate a number of entries associated with eachbucket of the group of buckets; determine, based on the comparison, asecond bucket of the group of buckets having a lower counter value; moveone or more entries associated with the virtual bucket from the firstbucket to the second bucket; and insert the entry into the secondbucket.

Example 2 may include the apparatus of example 1, wherein an entryincludes a key, a value, and a virtual bucket identifier.

Example 3 may include the apparatus of example 1, wherein move one ormore entries associated with the virtual bucket from the first bucket tothe second bucket further includes: determine a number of entries to bemoved; decrement a counter of the first bucket by the number of entriesto be moved; increment a counter of the second bucket by the number ofentries to be moved plus 1.

Example 4 may include the apparatus of any examples 1-3, wherein selectthe first bucket further includes selected the first bucket based uponentries in a mapping array.

Example 5 may include the apparatus of any examples 1-3, wherein thegroups of buckets correspondingly associated with the plurality ofvirtual buckets are identified in a mapping array.

Example 6 may include the apparatus of example 5, wherein the mappingarray is on the storage device.

Example 7 may include the apparatus of any one of examples 1-3, whereina data packet includes a network packet address.

Example 8 may include the apparatus of any one of examples 1-3, whereinto insert the entry comprises to manage a standard deviation of aplurality of counters of a respective plurality of buckets.

Example 9 may include the apparatus of any one of examples 1-3, whereinthe plurality of buckets is a fixed number.

Example 10 may include the apparatus of any one of examples 1-3, whereinthe number of entries in a bucket is limited to a fixed number.

Example 11 may include the apparatus of any one of examples 1-3, whereinto insert the entry is to be complete in O(1) operations.

Example 12 may include the apparatus of any one of examples 1-3, whereinto insert an entry comprises to manage the number of entriesrespectively in the plurality of buckets.

Example 13 may be a method for computing, comprising: receiving, by acomputing system, a data packet for insertion as an entry into a hashtable having a plurality of buckets divided into groups and the groupsof buckets correspondingly associated with a plurality of virtualbuckets; applying, by the computing system, a hash function to the datapacket to determine a virtual bucket associated with the entry;selecting, by the computing system, a first bucket of a group of bucketsassociated with the virtual bucket; comparing, by the computing system,respective counters of the group of buckets, wherein the respectivecounters indicate a number of entries associated with each bucket of thegroup of buckets; determining, by the computing system, based on thecomparison, a second bucket of the group of buckets having a lowercounter value; moving, by the computing system, one or more entriesassociated with the virtual bucket from the first bucket to the secondbucket; and inserting, by the computing system, the entry into thesecond bucket.

Example 14 may include the method of example 13, wherein moving one ormore entries associated with the virtual bucket from the first bucket tothe second bucket further includes: determining, by the computingsystem, a number of entries to be moved; determining, by the computingsystem, a counter of the first bucket by the number of entries to bemoved; incrementing, by the computing system, a counter of the secondbucket by the number of entries to be moved plus 1.

Example 15 may include the method of any one of examples 13-14, whereinselecting the first bucket further includes selecting the first bucketbased upon entries in a mapping array.

Example 16 may include the method of any one of examples 13-14, whereinthe groups of buckets correspondingly associated with the plurality ofvirtual buckets are identified in a mapping array.

Example 17 may include the method of example 16, wherein the mappingarray is on a storage device.

Example 18 may include the method of any one of examples 13-14, whereina data packet includes a network packet address.

Example 19 may include the method of any one of examples 13-14, whereininserting the entry comprises managing, by the computing system, astandard deviation of a plurality of counters of a respective pluralityof buckets.

Example 20 may include the method of any one of examples 13-14, whereinthe plurality of buckets is a fixed number.

Example 21 may include the method of any one of examples 13-14, whereinthe number of entries in a bucket is limited to a fixed number.

Example 22 may include the method of any one of examples 13-14, whereininserting the entry is to be complete in O(1) operations.

Example 23 may include the method of any one of examples 13-14, whereininserting the entry comprises managing, by the computing system, thenumber of entries respectively in the plurality of buckets.

Example 24 may be one or more computer-readable media comprisinginstructions that cause a computing device, in response to execution ofthe instructions by the computing device, to: receive, by the computingdevice, a data packet for insertion as an entry into a hash table havinga plurality of buckets divided into groups and the groups of bucketscorrespondingly associated with a plurality of virtual buckets; apply,by the computing device, a hash function to the data packet to determinea virtual bucket associated with the entry; select, by the computingdevice, a first bucket of a group of buckets associated with the virtualbucket; compare, by the computing device, respective counters of thegroup of buckets, wherein the respective counters indicate a number ofentries associated with each bucket of the group of buckets; determine,by the computing device, based on the comparison, a second bucket of thegroup of buckets having a lower counter value; move, by the computingdevice, one or more entries associated with the virtual bucket from thefirst bucket to the second bucket; and insert, by the computing device,the entry into the second bucket.

Example 25 may include the one or more computer-readable media ofexample 24, wherein move one or more entries associated with the virtualbucket from the first bucket to the second bucket includes: determine,by the computing device, a number of entries to be moved; determine, bythe computing device, a counter of the first bucket by the number ofentries to be moved; increment, by the computing device, a counter ofthe second bucket by the number of entries to be moved plus 1.

Example 26 may include the one or more computer-readable media of anyone of examples 24-25, wherein select the first bucket further includesselect the first bucket based upon entries in a mapping array.

Example 27 may include the one or more computer-readable media of anyone of examples 24-25, wherein the groups of buckets correspondinglyassociated with the plurality of virtual buckets are identified in amapping array.

Example 28 may include the one or more computer-readable media of claim27, wherein the mapping array is on a storage device.

Example 29 may include the one or more computer-readable media of anyone of examples 24-25, wherein a data packet includes a network packetaddress.

Example 30 may include the one or more computer-readable media of anyone of examples 24-25, wherein insert the entry comprises manage astandard deviation of a plurality of counters of a respective pluralityof buckets.

Example 31 may include the one or more computer-readable media of anyone of examples 24-25, wherein the plurality of buckets is a fixednumber.

Example 32 may include the one or more computer-readable media of anyone of examples 24-25, wherein the number of entries in a bucket islimited to a fixed number.

Example 33 may include the one or more computer-readable media of anyone of examples 24-25, wherein insert the entry is to be complete inO(1) operations.

Example 34 may include the one or more computer-readable media of anyone of examples 24-25, wherein insert the entry comprises manage thenumber of entries respectively in the plurality of buckets.

Example 35 may be an apparatus for computing, comprising: means forreceiving a data packet for insertion as an entry into a hash tablehaving a plurality of buckets divided into groups and the groups ofbuckets correspondingly associated with a plurality of virtual buckets;means for applying a hash function to the data packet to determine avirtual bucket associated with the entry; means for selecting a firstbucket of a group of buckets associated with the virtual bucket; meansfor comparing respective counters of the group of buckets, wherein therespective counters indicate a number of entries associated with eachbucket of the group of buckets; means for determining, based on thecomparison, a second bucket of the group of buckets having a lowercounter value; means for moving one or more entries associated with thevirtual bucket from the first bucket to the second bucket; and means forinserting the entry into the second bucket.

Example 36 may include the apparatus of example 35, wherein moving oneor more entries associated with the virtual bucket from the first bucketto the second bucket further includes: means for determining a number ofentries to be moved; means for determining a counter of the first bucketby the number of entries to be moved; means for incrementing a counterof the second bucket by the number of entries to be moved plus 1.

Example 37 may include the apparatus of any one of examples 35-36,wherein selecting the first bucket further includes means for selectingthe first bucket based upon entries in a mapping array.

Example 38 may include the apparatus of any one of examples 35-36,wherein the groups of buckets correspondingly associated with theplurality of virtual buckets are identified in a mapping array.

Example 39 may include the apparatus of example 38, wherein the mappingarray is on a storage device.

Example 40 may include the apparatus of any one of examples 35-36,wherein a data packet includes a network packet address.

Example 41 may include the apparatus of any one of examples 35-36,wherein inserting the entry comprises means for managing a standarddeviation of a plurality of counters of a respective plurality ofbuckets.

Example 42 may include the apparatus of any one of examples 35-36,wherein the plurality of buckets is a fixed number.

Example 43 may include the apparatus of any one of examples 35-36,wherein the number of entries in a bucket is limited to a fixed number.

Example 44 may include the apparatus of any one of examples 35-36,wherein inserting the entry is to be complete in O(1) operations.

Example 45 may include the apparatus of any one of examples 35-36,wherein inserting the entry comprises means for managing the number ofentries respectively in the plurality of buckets.

What is claimed is:
 1. An apparatus for computing, comprising: one ormore computer processors; a storage device coupled to the one or morecomputer processors; an insertion module communicatively coupled to theone or more processors, to manage insertion of entries into a hash tableresiding on the storage device, wherein the hash table has a pluralityof buckets to hold a plurality of entries, the buckets divided intogroups and the groups of buckets are correspondingly associated with aplurality of virtual buckets; and wherein the insertion module is to:receive a data packet for insertion as an entry into the hash table;apply a hash function to the data packet to determine a virtual bucketassociated with the entry; select a first bucket of the group of bucketsassociated with the virtual bucket; compare respective counters of thegroup of buckets, wherein the respective counters indicate a number ofentries associated with each bucket of the group of buckets; determine,based on the comparison, a second bucket of the group of buckets havinga lower counter value; move one or more entries associated with thevirtual bucket from the first bucket to the second bucket to balance thenumber of entries across the group of buckets; and insert the entry intothe second bucket.
 2. The apparatus of claim 1, wherein the entryincludes a key, a value, and a virtual bucket identifier.
 3. Theapparatus of claim 1, wherein move one or more entries associated withthe virtual bucket from the first bucket to the second bucket furtherincludes: determine a number of entries to be moved; decrement a counterof the first bucket by the number of entries to be moved; increment acounter of the second bucket by the number of entries to be movedplus
 1. 4. The apparatus of claim 1, wherein select the first bucketfurther includes selected the first bucket based upon entries in amapping array.
 5. The apparatus of claim 1, wherein the groups ofbuckets correspondingly associated with the plurality of virtual bucketsare identified in a mapping array.
 6. The apparatus of claim 5, whereinthe mapping array is on the storage device.
 7. The apparatus of claim 1,wherein the data packet includes a network packet address.
 8. Theapparatus of claim 1, wherein to insert the entry comprises to manage astandard deviation of a plurality of counters of a respective pluralityof buckets.
 9. The apparatus of claim 1, wherein the plurality ofbuckets is a fixed number.
 10. The apparatus of claim 1, wherein thenumber of entries in a bucket is limited to a fixed number.
 11. Theapparatus of claim 1, wherein to insert the entry is to be complete inO(1) operations.
 12. The apparatus of claim 1, wherein to insert theentry comprises to manage the number of entries respectively in theplurality of buckets.
 13. A method for computing, comprising: receiving,by a computing system, a data packet for insertion as an entry into ahash table having a plurality of buckets to hold a plurality of entries,the buckets divided into groups and the groups of bucketscorrespondingly associated with a plurality of virtual buckets;applying, by the computing system, a hash function to the data packet todetermine a virtual bucket associated with the entry; selecting, by thecomputing system, a first bucket of a group of buckets associated withthe virtual bucket; comparing, by the computing system, respectivecounters of the group of buckets, wherein the respective countersindicate a number of entries associated with each bucket of the group ofbuckets; determining, by the computing system, based on the comparison,a second bucket of the group of buckets having a lower counter value;moving, by the computing system, one or more entries associated with thevirtual bucket from the first bucket to the second bucket to balance thenumber of entries across the group of buckets; and inserting, by thecomputing system, the entry into the second bucket.
 14. The method ofclaim 13, wherein moving one or more entries associated with the virtualbucket from the first bucket to the second bucket further includes:determining, by the computing system, a number of entries to be moved;determining, by the computing system, a counter of the first bucket bythe number of entries to be moved; incrementing, by the computingsystem, a counter of the second bucket by the number of entries to bemoved plus
 1. 15. The method of claim 13, wherein selecting the firstbucket further includes selecting the first bucket based upon entries ina mapping array.
 16. The method of claim 13, wherein the groups ofbuckets correspondingly associated with the plurality of virtual bucketsare identified in a mapping array.
 17. The method of claim 16, whereinthe mapping array is on a storage device.
 18. The method of claim 13,wherein the data packet includes a network packet address.
 19. Themethod of claim 13, wherein inserting the entry comprises managing, bythe computing system, a standard deviation of a plurality of counters ofa respective plurality of buckets.
 20. The method of claim 13, whereinthe plurality of buckets is a fixed number.
 21. The method of claim 13,wherein the number of entries in a bucket is limited to a fixed number.22. The method of claim 13, wherein inserting the entry is to becomplete in O(1) operations.
 23. The method of claim 13, whereininserting the entry comprises managing, by the computing system, thenumber of entries respectively in the plurality of buckets.
 24. One ormore non-transitory computer-readable media comprising instructions thatcause a computing device, in response to execution of the instructionsby the computing device, to: receive, by the computing device, a datapacket for insertion as an entry into a hash table having a plurality ofbuckets to hold a plurality of entries, the buckets divided into groupsand the groups of buckets correspondingly associated with a plurality ofvirtual buckets; apply, by the computing device, a hash function to thedata packet to determine a virtual bucket associated with the entry;select, by the computing device, a first bucket of a group of bucketsassociated with the virtual bucket; compare, by the computing device,respective counters of the group of buckets, wherein the respectivecounters indicate a number of entries associated with each bucket of thegroup of buckets; determine, by the computing device, based on thecomparison, a second bucket of the group of buckets having a lowercounter value; move, by the computing device, one or more entriesassociated with the virtual bucket from the first bucket to the secondbucket to balance the number of entries across the group of buckets; andinsert, by the computing device, the entry into the second bucket. 25.The one or more non-transitory computer-readable media of claim 24,wherein move one or more entries associated with the virtual bucket fromthe first bucket to the second bucket includes: determine, by thecomputing device, a number of entries to be moved; determine, by thecomputing device, a counter of the first bucket by the number of entriesto be moved; increment, by the computing device, a counter of the secondbucket by the number of entries to be moved plus 1.